Test Reports are an essential part of a Selenium automation framework because they provide a clear summary of automated test execution. A test report shows which test cases passed, failed, were skipped, or encountered errors during execution.
In Selenium automation, test reports help QA engineers, developers, team leads, and stakeholders understand the health of an application without manually reviewing console output or source code. Reports can contain test names, execution status, execution duration, error messages, screenshots, logs, environment information, and other useful test evidence.
Test reports become especially important when a Selenium framework contains hundreds or thousands of automated test cases. Instead of checking every test manually, the team can review a centralized report and quickly identify failures and execution trends.
Course Resource: Selenium Training | Register for Course Demo
A Test Report is a document or web-based result generated after test execution that summarizes the outcome of test cases.
A typical report provides information such as:
Test reports convert raw automation execution results into information that can be easily understood and analyzed.
Test Cases
|
v
Selenium WebDriver
|
v
TestNG / PyTest
|
v
Test Execution
|
+----------------+
| |
v v
Passed Failed
| |
+-------+--------+
|
v
Test Report Generator
|
v
HTML / XML / JSON
|
v
Report Dashboard
|
v
QA / Developer / Stakeholder
Selenium WebDriver itself performs browser automation and does not provide a complete reporting framework. Selenium is generally combined with a test framework and reporting library to create detailed execution reports.
A common Selenium reporting architecture is:
Selenium WebDriver
+
TestNG
+
Reporting Library
|
v
Detailed Test Report
For example, a Java Selenium framework may use Selenium WebDriver for browser automation, TestNG for test execution, Maven for project management, and ExtentReports or Allure for reporting.
TestNG provides built-in reporting capabilities after test execution. When a TestNG suite is executed, TestNG generates result files that summarize test execution.
TestNG reports can provide information about:
TestNG reports are useful for basic execution analysis, while external reporting libraries can be used when richer dashboards and customization are required.
TestNG Suite
|
+-- Test
|
+-- Test Class
|
+-- Test Method 1
| |
| +-- PASS
|
+-- Test Method 2
| |
| +-- FAIL
|
+-- Test Method 3
|
+-- SKIP
import org.testng.Assert;
import org.testng.annotations.Test;
public class LoginTest {
@Test
public void validLoginTest() {
System.out.println("Valid login test");
Assert.assertTrue(true);
}
@Test
public void invalidLoginTest() {
System.out.println("Invalid login test");
Assert.assertTrue(true);
}
}
After execution, TestNG records the result of each test method and includes it in the generated execution reports.
Test reports normally categorize test executions into different statuses.
| Status | Description |
|---|---|
| PASS | The test executed successfully and all required validations passed. |
| FAIL | The test execution completed with a failed assertion or error. |
| SKIP | The test was not executed because it was skipped or depended on another failed test. |
| ERROR | An unexpected execution problem occurred outside the normal assertion flow. |
A passed test indicates that the test completed successfully according to its defined validations.
LoginTest
|
+-- validLoginTest
|
+-- PASS
A report can display the test name, execution time, status, and other available information.
A failed test indicates that an expected condition was not satisfied or an exception occurred during execution.
LoginTest
|
+-- invalidLoginTest
|
+-- FAIL
|
+-- AssertionError
|
+-- Screenshot
|
+-- Stack Trace
A detailed failure report helps the team understand where and why the automation test failed.
A skipped test is a test that was not executed. TestNG can skip tests because of explicit configuration, dependencies, or other execution conditions.
LoginTest
|
+-- loginTest
|
+-- SKIPPED
|
+-- Reason
HTML reports are widely used because they can be opened in a browser and presented in a visually understandable format.
An HTML report may contain:
Automation Test Report
--------------------------------
Total Tests : 20
Passed : 16
Failed : 3
Skipped : 1
Execution Time: 04:35
Test Results
--------------------------------
Login Test PASS
Search Test PASS
Cart Test PASS
Checkout Test FAIL
Profile Test PASS
ExtentReports is a popular reporting library used with Java automation frameworks. It can generate detailed HTML reports containing test statuses, logs, screenshots, and execution information.
ExtentReports can be integrated with Selenium and TestNG to produce customized and visually rich reports.
A typical architecture is:
TestNG
|
v
Selenium Test
|
v
ExtentReports
|
+-- Test Status
+-- Logs
+-- Screenshots
+-- Exceptions
+-- Environment
|
v
HTML Report
In a Maven-based Java project, a reporting library can be added through the project's dependency configuration. The exact dependency version should be selected according to the framework version and project requirements.
<dependency>
<groupId>com.aventstack</groupId>
<artifactId>extentreports</artifactId>
<version>YOUR_VERSION</version>
</dependency>
Using a project-specific version is recommended rather than blindly copying an outdated dependency version.
ExtentReports extent = new ExtentReports();
ExtentSparkReporter reporter =
new ExtentSparkReporter("test-output/report.html");
extent.attachReporter(reporter);
The reporter is configured with the location where the generated HTML report will be stored.
ExtentTest test = extent.createTest("Login Test");
test.info("Starting login test");
test.pass("Login test completed successfully");
The ExtentTest object is used to add information, status messages, warnings, failures, and other execution details.
After test execution, the report should be flushed so that the collected information is written to the report file.
extent.flush();Without properly flushing the report, the generated report may not contain all collected execution information.
import com.aventstack.extentreports.ExtentReports;
import com.aventstack.extentreports.ExtentTest;
import com.aventstack.extentreports.reporter.ExtentSparkReporter;
public class ReportExample {
public static void main(String[] args) {
ExtentReports extent = new ExtentReports();
ExtentSparkReporter reporter =
new ExtentSparkReporter(
"test-output/automation-report.html");
extent.attachReporter(reporter);
ExtentTest test =
extent.createTest("Login Test");
test.info("Opening application");
test.info("Entering username");
test.info("Entering password");
test.pass("Login successful");
extent.flush();
}
}
Detailed test-step logging makes reports easier to understand.
test.info("Opening login page");
test.info("Entering username");
test.info("Entering password");
test.info("Clicking login button");
test.pass("User successfully logged in");
When a failure occurs, logging helps identify the last successfully executed step.
A reporting framework can record failure information along with the failed test status.
try {
test.info("Executing login test");
// Selenium test steps
test.pass("Login successful");
} catch (Exception e) {
test.fail("Login test failed");
test.fail(e.getMessage());
throw e;
}
The exact reporting implementation should be designed so that the original test failure is still propagated to the test framework.
Screenshots are extremely useful when a Selenium test fails. A screenshot can show the state of the browser at the time of failure.
Typical flow:
Test Failure
|
v
Capture Screenshot
|
v
Save Screenshot
|
v
Attach to Report
|
v
Review Failure Evidence
File source =
((TakesScreenshot) driver)
.getScreenshotAs(OutputType.FILE);
Files.copy(
source.toPath(),
Paths.get("test-output/login-failure.png"),
StandardCopyOption.REPLACE_EXISTING
);
The screenshot can then be attached to a reporting framework using the reporting library's supported media or attachment mechanism.
A common framework design is to capture a screenshot automatically whenever a test fails.
Test Execution
|
v
Test Passed? ---- Yes ----> PASS Report
|
No
|
v
Capture Screenshot
|
v
Capture Error
|
v
Attach Evidence
|
v
FAIL Report
ITestListener is a TestNG listener interface that can be used to respond to test execution events.
It is commonly used for framework-level reporting, logging, screenshots, and execution monitoring.
Important listener methods include:
import org.testng.ITestListener;
import org.testng.ITestResult;
public class TestListener implements ITestListener {
@Override
public void onTestStart(ITestResult result) {
System.out.println(
"Test Started: " +
result.getName());
}
@Override
public void onTestSuccess(ITestResult result) {
System.out.println(
"Test Passed: " +
result.getName());
}
@Override
public void onTestFailure(ITestResult result) {
System.out.println(
"Test Failed: " +
result.getName());
}
@Override
public void onTestSkipped(ITestResult result) {
System.out.println(
"Test Skipped: " +
result.getName());
}
}
A TestNG listener can be registered in different ways depending on the framework architecture. One common approach is using the @Listeners annotation.
import org.testng.annotations.Listeners;
@Listeners(TestListener.class)
public class LoginTest {
@Test
public void loginTest() {
System.out.println("Login Test");
}
}
Listeners can also be configured through TestNG XML in larger automation frameworks.
The onTestSuccess() method is called when a TestNG test method completes successfully.
@Override
public void onTestSuccess(ITestResult result) {
System.out.println(
"PASS: " + result.getName());
}
This event can be used to add a pass status to a custom report.
The onTestFailure() method is useful for failure handling.
@Override
public void onTestFailure(ITestResult result) {
System.out.println(
"FAIL: " + result.getName());
}
A Selenium framework can use this event to capture screenshots, browser information, exception details, and logs.
The onTestSkipped() method can be used to record skipped test executions.
@Override
public void onTestSkipped(ITestResult result) {
System.out.println(
"SKIPPED: " + result.getName());
}
Allure is another reporting solution commonly used with automated testing frameworks. It provides a web-based interface for analyzing test execution results.
Allure reports can present:
TestNG / Selenium
|
v
Test Execution
|
v
Allure Results
|
v
Allure Report Generation
|
v
Web-Based Report
Attachments provide additional evidence for a test execution.
Common report attachments include:
Attachments should be selected carefully so reports remain useful without exposing sensitive information.
A professional report should provide a high-level summary before detailed test information.
| Metric | Example |
|---|---|
| Total Tests | 100 |
| Passed | 85 |
| Failed | 10 |
| Skipped | 5 |
| Pass Percentage | 85% |
| Execution Duration | 25 minutes |
A simple pass percentage can be calculated using:
Pass Percentage =
(Passed Tests / Total Executed Tests) × 100
For example:
Passed Tests = 85
Total Tests = 100
Pass Percentage =
(85 / 100) × 100
= 85%
When reporting metrics, teams should define clearly whether skipped tests are included in the denominator.
Test duration indicates how long an individual test or complete suite took to execute.
Duration information can help identify:
Cross-browser Selenium frameworks can include browser information in reports.
| Test | Browser | Status |
|---|---|---|
| Login Test | Chrome | PASS |
| Login Test | Firefox | PASS |
| Login Test | Edge | FAIL |
This makes browser-specific failures easier to identify.
Reports can include environment details such as:
Example:
Environment : QA
Browser : Chrome
OS : Windows
Java : 17
Framework : TestNG
Build : 1024
URL : https://qa.example.com
Data-driven tests can generate multiple test invocations. Reports should make it possible to identify which data set was used for each execution.
Login Test
|
|-- admin / valid password
| +-- PASS
|
|-- manager / valid password
| +-- PASS
|
|-- invalid / wrong password
+-- FAIL
Sensitive values such as passwords should not be displayed in plain text in reports.
Test reports work well with the Page Object Model because the framework can log high-level test actions while page classes handle Selenium interactions.
Test Method
|
v
Report Log
|
v
Page Object
|
v
Selenium WebDriver
|
v
Application
For example, a test can report "User login started" while the LoginPage class handles the actual element interactions.
test.info("Opening login page");
loginPage.enterUsername(username);
test.info("Username entered");
loginPage.enterPassword(password);
test.info("Password entered");
loginPage.clickLogin();
test.info("Login button clicked");
test.pass("Login workflow completed");
This creates a readable sequence of test activities in the report.
Assertions verify whether the actual application behavior matches the expected behavior.
String actualTitle = driver.getTitle();
String expectedTitle = "Dashboard";
Assert.assertEquals(actualTitle, expectedTitle);
A reporting framework can record the result of this assertion and, when appropriate, capture failure evidence.
Maven is commonly used to build Selenium TestNG projects and execute automated tests.
mvn testA typical execution flow is:
Maven
|
v
Compile Project
|
v
Run TestNG
|
v
Execute Selenium Tests
|
v
Generate Test Results
|
v
Generate Reports
Automated reports are particularly useful in CI/CD pipelines because tests can execute automatically after code changes.
Developer Commit
|
v
Git Repository
|
v
Jenkins / CI Pipeline
|
v
Maven Build
|
v
TestNG
|
v
Selenium Tests
|
v
Test Results
|
v
HTML Report
|
v
Team Notification
Jenkins can execute automated Selenium tests and publish or archive test results depending on the reporting and pipeline configuration.
A typical Jenkins flow is:
When tests run in CI environments, screenshots become particularly useful because the tester may not be watching the browser directly.
CI Test Failure
|
v
Capture Screenshot
|
v
Save Artifact
|
v
Attach to Report
|
v
Review Failure Remotely
Logging and reporting are related but serve different purposes.
| Logging | Reporting |
|---|---|
| Records application/test events | Summarizes test execution results |
| Useful for debugging | Useful for result analysis |
| Can contain detailed technical messages | Usually presents structured test information |
| Often generated continuously | Usually reviewed after or during execution |
| Examples include Log4j logs | Examples include TestNG, ExtentReports, Allure reports |
| Console Output | Test Report |
|---|---|
| Mostly text based | Can provide structured visual information |
| Harder to review after long execution | Easier to analyze |
| Limited organization | Can include dashboards and sections |
| Temporary unless captured | Can be stored as build artifacts |
| Limited screenshots | Can include screenshots and attachments |
A custom reporting framework can be created to standardize how automation results are presented.
A custom report may contain:
project
|
|-- src
| |-- main
| |-- test
|
|-- test-output
| |-- reports
| | |-- automation-report.html
| |
| |-- screenshots
| | |-- login-failure.png
| | |-- checkout-failure.png
| |
| |-- logs
| |-- automation.log
|
|-- pom.xml
|-- testng.xml
Report filenames should be meaningful and consistent.
Examples:
selenium-test-report.html
regression-report.html
smoke-test-report.html
build-1024-report.html
For CI environments, build numbers or execution timestamps can be incorporated into report filenames when appropriate.
Regression testing can generate large numbers of test results. A regression report helps the team understand whether previously working functionality continues to behave as expected.
| Module | Total | Passed | Failed | Skipped |
|---|---|---|---|---|
| Login | 20 | 19 | 1 | 0 |
| Search | 15 | 14 | 1 | 0 |
| Cart | 25 | 23 | 2 | 0 |
| Checkout | 20 | 17 | 2 | 1 |
Smoke tests verify whether the major application functionality is stable enough for further testing.
A smoke report can provide a quick overview:
Smoke Suite
--------------------------------
Login PASS
Search PASS
Cart PASS
Checkout PASS
Logout PASS
--------------------------------
Result: PASS
A good report should make failure investigation easier.
A failure analysis section can include:
Screenshots may contain confidential information such as customer information, account numbers, internal URLs, tokens, or other sensitive application data.
Before attaching screenshots to reports, consider:
Parallel Selenium execution can improve execution time, but the reporting framework must also be designed for concurrent test execution.
Test Suite
|
+---- Thread 1 ----> Test A ----> Report
|
+---- Thread 2 ----> Test B ----> Report
|
+---- Thread 3 ----> Test C ----> Report
|
+---- Thread 4 ----> Test D ----> Report
|
v
Combined Results
Thread-safe framework design is important when multiple tests write to the same reporting infrastructure.
Reports can categorize failures to make analysis easier.
Possible categories include:
The classification should be based on the actual investigation rather than automatically assuming that every failed automation test represents an application defect.
Keeping historical test reports can help teams compare test execution results across builds.
Build 100
|
+-- 92% Passed
|
Build 101
|
+-- 94% Passed
|
Build 102
|
+-- 89% Passed
|
Build 103
|
+-- 96% Passed
Historical data can help teams investigate changes in test stability and identify recurring failures.
Selenium Test Automation
|
v
TestNG Framework
|
+-------------+-------------+
| |
v v
Test Execution ITestListener
| |
+-------------+-------------+
|
v
Reporting Manager
|
+-------------+-------------+
| | |
v v v
Logs Screenshots Test Status
| | |
+-------------+-------------+
|
v
Extent / Allure
|
v
HTML Report
|
v
CI/CD Build Artifact
|
v
Team / Stakeholders
Large Selenium frameworks can centralize report creation in a dedicated reporting utility.
public class ReportManager {
private static ExtentReports extent;
public static ExtentReports getReportInstance() {
if (extent == null) {
extent = new ExtentReports();
ExtentSparkReporter reporter =
new ExtentSparkReporter(
"test-output/report.html");
extent.attachReporter(reporter);
}
return extent;
}
}
In production frameworks, thread safety, lifecycle management, report naming, and parallel execution should be handled according to the framework architecture.
import com.aventstack.extentreports.ExtentReports;
import com.aventstack.extentreports.ExtentTest;
import com.aventstack.extentreports.reporter.ExtentSparkReporter;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class LoginTest {
WebDriver driver;
static ExtentReports extent;
ExtentTest test;
@BeforeMethod
public void setup() {
driver = new ChromeDriver();
driver.manage().window().maximize();
if (extent == null) {
extent = new ExtentReports();
ExtentSparkReporter reporter =
new ExtentSparkReporter(
"test-output/login-report.html");
extent.attachReporter(reporter);
}
}
@Test
public void loginTest() {
test = extent.createTest("Login Test");
test.info("Opening application");
driver.get("https://example.com/login");
test.info("Entering username");
driver.findElement(By.id("username"))
.sendKeys("testuser");
test.info("Entering password");
driver.findElement(By.id("password"))
.sendKeys("testpassword");
test.info("Clicking login button");
driver.findElement(By.id("loginButton"))
.click();
String title = driver.getTitle();
test.info("Validating page title");
Assert.assertTrue(
title.contains("Dashboard"));
test.pass("Login test passed");
}
@AfterMethod
public void tearDown() {
if (driver != null) {
driver.quit();
}
if (extent != null) {
extent.flush();
}
}
}
This example demonstrates the basic relationship between Selenium, TestNG, assertions, and an HTML reporting library.
When a Data Provider is used, each data set can represent a separate test invocation.
@DataProvider(name = "users")
public Object[][] users() {
return new Object[][] {
{"admin"},
{"manager"},
{"employee"}
};
}
@Test(dataProvider = "users")
public void loginTest(String username) {
System.out.println(
"Testing user: " + username);
}
A detailed reporting framework can record the test name and relevant non-sensitive data for each invocation.
An e-commerce automation framework may generate reports for:
Example:
E-Commerce Regression Report
Login PASS
Product Search PASS
Product Filter PASS
Product Details PASS
Add to Cart PASS
Cart Validation FAIL
Checkout PASS
Order Confirmation PASS
Logout PASS
Test reports can help QA teams identify failed scenarios that may require further investigation. A report should provide enough evidence to reproduce or analyze a failure.
A typical defect investigation may use:
Failed Test
|
+-- Test Name
+-- Build Number
+-- Environment
+-- Browser
+-- Error Message
+-- Stack Trace
+-- Screenshot
+-- Logs
|
v
Defect Investigation
Automation reports can accumulate quickly in CI environments. A project should define how long reports, screenshots, logs, and other artifacts should be retained.
Good practices include:
A test report summarizes the results and relevant evidence of automated or manual test execution.
They provide a structured view of automation results and help teams analyze failures, execution status, and test evidence.
Selenium WebDriver focuses on browser automation. Detailed reporting is generally provided by the test framework and reporting libraries integrated with the Selenium framework.
TestNG manages test execution and provides execution results that can be consumed by reporting systems.
ExtentReports is a reporting library used to create detailed and customizable HTML reports for automated tests.
Allure is a reporting solution that provides a web-based interface for analyzing automated test results, steps, attachments, and failures.
ITestListener is a TestNG listener interface that allows automation frameworks to react to test execution events such as start, success, failure, and skip.
A Selenium screenshot can be captured using TakesScreenshot and then attached to the reporting framework using its supported attachment mechanism.
They provide visual evidence of the browser state when a test fails.
Logging records technical events and messages, while reporting presents structured test execution results and evidence.
It should normally contain test status, test names, execution duration, failure details, environment information, and useful evidence such as screenshots and logs.
Yes. Selenium TestNG reports can be generated during CI/CD execution and stored or published as build artifacts according to the pipeline configuration.
Report aggregation combines results from multiple tests, classes, suites, browsers, environments, or parallel executions into a consolidated report.
Passwords and other secrets are sensitive information and should not be unnecessarily exposed in logs or reports.
It is a report entry showing that a test did not satisfy its expected condition or encountered an execution error.
A skipped test is a test that was not executed because of an execution condition, dependency, configuration, or explicit skip behavior.
Browser information helps identify browser-specific failures in cross-browser automation.
Flushing writes the collected reporting information to the configured report output.
Yes. TestNG can represent individual Data Provider invocations as separate test executions, depending on the reporting integration.
A useful report is clear, searchable, sufficiently detailed for failure investigation, and free from unnecessary sensitive information.
| Concept | Description |
|---|---|
| Test Report | Summary of test execution results and evidence. |
| TestNG | Java testing framework used to execute and manage tests. |
| ExtentReports | Reporting library for detailed HTML test reports. |
| Allure | Web-based test reporting solution. |
| ITestListener | TestNG listener for test execution events. |
| Screenshot | Visual evidence of browser state. |
| Log | Technical information recorded during execution. |
| PASS | Test completed successfully. |
| FAIL | Test did not meet the expected result or encountered an error. |
| SKIP | Test was not executed. |
| CI/CD | Automated build and test execution environment. |
Consider an e-commerce application where a QA automation team executes login, search, product, cart, checkout, and logout tests.
E-Commerce Automation
|
v
TestNG Suite
|
+-- Login Tests
+-- Search Tests
+-- Product Tests
+-- Cart Tests
+-- Checkout Tests
+-- Logout Tests
|
v
Selenium WebDriver
|
v
Assertions
|
v
ITestListener
|
+-- Pass
+-- Fail
+-- Skip
+-- Screenshot
+-- Logs
|
v
ExtentReports / Allure
|
v
HTML Report
|
v
CI/CD Pipeline
Test Reports are a critical component of professional Selenium automation frameworks. They convert test execution information into a structured format that can be easily reviewed by QA engineers, developers, automation engineers, and stakeholders.
Selenium WebDriver performs browser automation, while TestNG manages test execution and reporting integrations can provide detailed HTML-based results, logs, screenshots, and other evidence.
Popular reporting approaches include TestNG's built-in results, ExtentReports, Allure, and custom reporting solutions. TestNG listeners such as ITestListener can be used to automatically handle test-start, success, failure, and skip events.
For a production-quality Selenium framework, reporting can be integrated with Page Object Model, Data Providers, Maven, screenshots, logging, cross-browser execution, parallel execution, and CI/CD pipelines.
A well-designed report should be clear, useful for debugging, easy to access, and careful not to expose sensitive information such as passwords, tokens, or confidential application data.
Learn more about Selenium automation, TestNG, reporting, frameworks, and practical testing:
Final Takeaway: Test Reports make Selenium automation results understandable, traceable, and useful for debugging and decision-making. By combining TestNG execution, listeners, screenshots, logs, and reporting tools such as ExtentReports or Allure, automation teams can create a professional reporting system that supports local testing as well as CI/CD-based execution.
